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ABSTRACT 

The Napier University Business School, Edinburgh, 
Scotland and the Katholieke Universiteit, Leuven, Belgium embarked on 
a 3-year collaborative research project to evaluate lecturing 
techniques at Napier and to compare the results of the evaluation at 
Napier with those achieved at Leuven. Leuven had developed the EVADOC 
Questionnaire to monitor lecture method teaching quality in the 
1970s. This questionnaire focuses on clarification of instructional 
aims and objectives, recognition of students* backgrounds, 
preparation (content, explanation, media), interaction, and 
evaluation of assessment. These are combined with student perceptions 
of teachers * directing, organizing, communication, stimulating, and 
motivating behaviors. During the pilot year at Napier eight lecturers 
took part in the study and distributed the questionnaire to their 
students. The completed answer sheets were then sent to Leuven for 
analysis. Results confirmed the validity of the questionnaire for 
evaluating lecturing behavior at Napier. As a consequence, the 3-year 
project was undertaken and the first year completed with 9 lecturers 
taking part and A3A student responses in April 1993. The project also 
provided a seminar for interested staff and private interviews with 
lecturers to provide feedback. In addition the project initiated a 
pattern of two visits to Leuven a year to compare results. (JB) 
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1. INTRODUCTION 

In Session 1992-93 Napier University Business School, Edinburgh, Scotland tux] the 
Katholieke Universiteit, Leuven, Belgium embarked on a three-year collaborative 
research project to evaluate lecturing techniques at Napier and to compare the resuhs 
of the evaluation at Napier with those achieved at Leuven, bearing in mind that these 
institutions exist in two very different educational cultures. 

2. EXISTING RESEARCH INTO THE EVALUATION OF LECTURING BEHAVIOUR 
AND TECHNIOUES 

Research into evaluating lecturing behaviour and techniques has been ongoing for 
many years and there have been many publications on this subject - Lowyck 1975; 
Cohen P A 1980 and De Neve and Janssen 1982 to name but three. Research in this 
area began at Leuven in 1975 and, following the establisliment of an Educational 
Services Research Unit at Leuven in 1977, this research into lecturing techniques was 
led by Professor Hubert de Neve. The use of a questionnaire (EVADOC) based on 
Van Gelder's model for didactic analysis was found to be a sound tool for carrying 
out such an evaluation and has now been used for many years at Leuven. 

3. INITIATING COLLABORATIVE RESEARCH 

3.1 Establishing Links with the Katholieke Universiteit Leuven 

Following an initial approach to the British Council in August 1990 regarding its 
British-Flemish Academic Research Collaboration Progranuuc contact was made with 
Professor Hubert de Neve of Leuven University who was involved in research in areas 
similar to our own research interests. 

3.2 Explorator\' Visits 

Two exploratory visits were made to the Katholieke Universiteit, Leuven^ in February 
and November 1991 during which the basis of a pilot study into lecturing behaviour 
and techniques at Napier University was established. Those involved at .his stage of 
the research project were Professor Hubert de Neve and Professor Herman Verhaeghe 
of the Educational Services Research Unit at Leuven and Professor Jeff McLachlan 
and Mrs Vivienne Wood of the Business School at Napier University. However, 
Professor de Neve left the University in March 1992 and his role in the Project was 
taken over by Professor Piet Janssen. Director of the Educational Sendees Research 
Unit. 

4. OBJECTIVES OF THE RESEARCH PROJECT 

During the preliminary discussions it was agreed that, despite the differences between 
the educational cultures of Belgium and the United Kingdom, it would be useful to 
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test the questionnaire and methods used at Leuven for evaluating lecturing f^ehaviour 
and technique in another educational culture and to endeavour to draw comparisons 
between the results achieved in Leuven and Scotland, following the conduct of a 
formal three year project. 

Given the impending change of status of Napier from a Polytechnic to a University, 
and the possible implications of the establishment of a Scottish Higher Education 
Funding Council in terms of accountability and quality of delivery, it was recognised 
that the opportunity to carry out and test such a system, would be highly 
advantageous. 

The establishment of a pilot project, to test the validity of the questionnaire, modified 
for use in Napier, was seen to be an essential pre-requisite to the establishment of the 
formal three year project. 

5. CONSTRUCTION OF THE EVADOC QUESTIONNAIRE BY LEUVEN 
UNIVERSITY 

5.1 The Need for Lecmring Evaluation 

As a result of the 'open access' to the University system operated in Belgium and the 
ever increasing first year enrolments the need for high quality teachers who could 
cope efficiently with the large sized classes became cmcial and in response to requests 
from academic staff and students for steps to be taken to stimulate interest in the 
quality of teaching the Educational Services Research Unit was established in 1977. 

Interest in the techniques of lecturing evaluation had already been fostered in 1975 
with attention being paid to the evaluation systems already in force in the United 
States, Canada and Australia but many of the instruments used were rather complex 
and involved lengthy processes for completion of questionnaires and the analysis of 
the results. 

In 1978 Leuven considered the use of American didactic factor analysis methods 
(Menges 1973) conceding that if relevant feedback was to be obtained then only those 
statements/questions that would have such relevance in the eyes of didactic experts 
would be acceptable. The resulting EVADOC Questionnaire constructed by Leuven 
was based on Van Gelder's model for didactic analysis (Van Gclder et al: 
1971:1972:1974). 

5.2 Structuring the EVADOC Questionnaire 

5.2.1 Aspects of Lecturing Behaviour 

The EVADOC Questionnaire was constmcted to assess the quality of 
lecturing behaviour, by means of student evaluation, with a view to 
identifying areas of weakness requiring improvement and areas of 
excellence that could serve as best practice. 

/ 
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The EVADOC questionnaire consists of 97 statements in relation to 
seven specific observable aspects of teaching activity /technique 
identified by Glaser (1962) and Van Gelder (1975) which are 
considered to be essential for high quality teaching. 

These aspects of teaching activity/technique are: 

(a) Clarification of instructional aims and objectives: Does the 
lecturer clarify (i) his own objectives in relation to the areas he 
will cover and for what purpose; (ii) the relevajice of his 
subject within the course as a whole and the way in which it 
integrates with the other subjects of the course. 

(b) Recognition of Students' Backgrounds: Does the lecturer 
identify and build upon the students' prior knowledge and 
experience and their interests linking these factors to the 
material being taught? 

(c) Preparation 

(i) Content Is the content of his lectures carefully 
prepared including the selection of relevant, interesting 
and topical materials? 

(ii) Explanation Has the lecturer prepared clear, structured 
and concise explanations of the topics to be covered? 

(iii) Media Does the lecturer make good use of 
teaching aids such as blackboards, audio visual aids, 
course materials etc? 

(d) Interaction: Does the lecturer interact with the students by 
creating opportunities for question/answer sessions, discussion, 
exchange of ideas etc? 

(e) Evaluation of Assessment : Does the lecturer make it clear to 
the students what they are expected to learn if they aix^ to 
succeed in assessments/examinations and does he explain the 
format of the assessments, familiarising the student with the 
type of questions they should expect? 

These seven components of teaching activity/technique fomied the 
general framework of the questionnaire. However, as a result of 
research being conducted elsewhere (Lowyck J 1975) which had shown 
that students evaluate teaching activities/techniques in terms of their 
perceptions of the way in which they facilitates their studying 
behaviour, their ability to learn, understand, analyse and apply the 
knowledge and thus to perform well and to progress to the next stage 
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of their course, a factor analysis of the students* responses to tlie 
questionnaire was carried out (De Neve and Janssen 1982). This factor 
analysis revealed five major factors or perception dimensions that, from 
a student's point of view indicate good teaching technique. These 
same five factors were identified in a number of studies of different 
questionnaires, constructed to evaluate different teaching formats, and 
evidence of their consistency was found in a survey conducted by 
Marsh in 1987, which led to the conclusion that these factors, or 
perception dimensions, indicated the aspects of teaching 
activities/techniques that students would wish to see fiilfilled. 

5.2.2 The Five Perception Dimensions of Good Lecturing Behaviour 

The five perception dimensions of good lecturing behav;»mr identified 
are as follows: 

(a) Directing (Guiding) Behaviour 

When a lecturer directs the students' study behaviour he guides 
them towards achieving future objectives/goals, ie passing the 
examinationy^rogressing to the next stage of the course. He wUl 
do this by helping them to prepare for the examination/ 
assessments by clarifying the requirements of the 
examination/assessments; stressing the important aspects of the 
course; alerting the students to difficult areas of the syllabus 
and preparing students for the stracture of the 
examination/assessment in question. 

(b) Organising Behaviour 

Well organised lecture behaviour enables the student to 
understand the information being provided in a logical and 
coherent pattern which is clearly the first prerequisite for a 
student to be able to assimilate knowledge. An organised 
lecturer will create a clear stracture for his/lier lectures and will 
make good use of visual aids. 

(c) Communicating Behaviour 

Good communication skUls aid the growing confidence of the 
student and can be experienced in a number of teaching 
behaviours. 

Students expect regular support and feedback to tell them 
whether they are going in the right direction or not and this 
depends on the lecturer's ability to communicate effectively 
with the students. This can be made easier if the lectuier is 
open to question and discussion and is ready to listen to the 
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opinions of the students. However, good communication skills 
are only effective in so far as the other diinensions of good 
teaching behaviour are present, ie directing; orgmiising; 
stimulating and motivating. 

(d) Stimulating Behaviour 

The more attention a lecturer pays to the meaning of his course, 
the greater is the chance that the material used in his lectures 
will be perceived by the students as providing an enriching 
experience in which learning and real life have become 
integrated. As a result the students will feel confident in theii' 
understanding of the subject and in control of their 
learning/study pattern, being able to relate new facts and 
information to their previous knowledge. Evidence of 
stimulating behaviour will include discussion, the use of 
interesting and relevant examples which enliven lectures and 
systematic explanations of where new topics fit into what has 
already been learned. 

(e) Motivating Behaviour 

Motivating or inspiring behaviour implies that the lecturer 
involves the students in their own learning process by 
deepening their interest in the subject. 

The student will feel that he is personally involved m his 
studying and will feel inspired to undertake a thorough analysis 
of the material before hun which contributes to the deep level 
thinking approach so necessary for effective studying. 

Here the lecturer will be genuinely interested in his/her subject, 
presenting subject material in a clearly understandable fommt 
and encouraging the students to think of the subject in a wider 
context. 

It was therefore necessary to combine these dimensions of teaching 
activities/techniques with the students' perception dimensions if an 
accurate evaluation of lecturing behaviour was to be carried out. 

6. METHODOLOGY FOR ANALYSING QUESTIONNAIRE RESULTS 

6.1 Matrix Approach 

On the one hand the structure of the questionnaire was based on a theoretical 
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description of *good lecturing* using specific observable aspects of teaching 
activities/techniques while on the other hand analysis of the questionnaire results in 
terms of the five perception scores indicated how well a lecturer was facilitating the 
students' study behaviour. 

By representing the Questionnaire in a carefully designed matrix based on the student 
perception dimension (columns of the matrix) and an educational component or 
teaching activities/techniques dimension (rows of the matrix), each cell of the matrix 
represents a specific lecturing strategy which can be interpreted as relevant from an 
educational point of view and as an opponunity to facilitate studying behaviour. 

6.2 Completion and Analysis of the Questionnaire 

Students are asked to express the extent to which they agree or disagree using a six- 
point scale, with the statements formulated in relation to the observable lecturing 
behaviours of the lecturer concerned. Their answers are recorded on optically read 
sheets the data from which is analysed by a computer program developed by Leuven. 
In using a six point scale the point which differentiates an acceptable score from an 
unacceptable score is 3.5. The programme calculates the mean scores for the five 
perception dimensions, the seven teaching activites/techniques dimensions and for each 
cell of the matrix both in terms of the mean of all of the lecturers' scores and of the 
mean of the individual lecturer concerned. This enables each lecturer to see their 
own level of scoring in terms of absolute scores and in terms of their relative position 
compared with their colleagues. 

7. FEEDBACK AND INTERPRETATION OF RESULTS 

The feedback obtained in relation to each lecturer is kept totally confidential and is 
known only to those carrying out the evaluation and to the individual lecturer 
concerned. Feedback is presented in a written format and is discussed and interpreted 
in a face to face interview between the parties concerned. To be successful, praise 
should be given where areas of excellence are identified and the lecmrer should be 
encouraged to discuss his techniques with colleagues to ensure the spread of best 
practice. 

Where areas of weakness are identified it is better to select one or two iu*e:is at most 
to be addressed by the lecmrer over the coming year. Personal advice can be given 
as to how the specific aspects of lecturing behaviour can be improved and this can 
include referring the lecturer to a colleague who is known to have strengths in tlie area 
m question. Particular attention should then be paid to the area where weakness was 
identified when the lecturer is again evaluated. Lecturers experience a real sense of 
acliievement and satisfaction when they obtain higher scores in a subseciuent 
evaluation for an art a previously identified as having some weakness. 

In the past at Leuven only those lecturers asking to be evaluated have had their 
lecturing behaviour analysed by means of the EVADOC Questionnaire. However, a 
new policy on appointments of academic staff, both newcomers and for promotions, 
has recently been introduced which requires that candidates must not only give 
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evidence of their worth as a researcher but must also give a presentation of their 
teaching dossier. Evidence of having undergone the EVADOC evaluation is seen to 
be an advantage particularly as there is now a shift in policy at Leuven away from the 
main focus being placed on research skills to the need for quality teaching skills to 
be present as well. 

ADAPTATION OF THE EVADOC QUESTIONNAIRE FOR USE AT NAPIER 
UNIVERSITY 

Following the two exploratory visits to Leuven in 1991 the Flemish questionnaire was 
translated at Napier University and the questions were modified to suit the Scouish 
education system and the particular requirements of Napier University. The diaft 
translation and modifications were assessed by Leuven and, following clarification of 
a number of areas relating to the education culture in Scotland, it was agreed that th ." 
modified questionnaire would be an appropriate instmment for carrying out the pilot 
evaluation of lecturing techniques at Napier. By conducting a pilot evaluation and by 
analysing the responses received, the validity of the modified questionnaire could then 
be tested and assessed. 

The resulting EVALEC questionnaire, (EVAluation of LECturing teclinique), and 
bearing the logos of both Universities, was printed and reproduced at Napier for use 
in the pilot study. 

CONDUCT OF THE PILOT STUDY 
Completion of the EVALEC Questionnaire 

Eight lecturers in the Napier Business School volunteered to take part in the pilot 
study in 1992. All of the lecturers taught first year students on degree and Higher 
National Diploma courses in the Business School. The questionnaires, together with 
^e optically read answer sheets provided by Leuven, were distributed to the lecturers 
concerned at the beginning of the summer term for completion by students, in class 
time, in the week beginning 20 AprU. The completed answer sheets - 191 in total - 
were coded to ensure that the resulting data was attributed to the appropriate lecturer 
and sent to Leuven in mid-May for analysis by a graduate student in education - 
Sabine De Valck - who was carrying out the analysis of the data for her Masters 
Thesis, under the guidance of Professor Piet Janssen. There followed an exchange of 
correspondence, mainly by Fax, regarding clarification of a number of issues relating 
to the status and funding of Napier; the qualifications and ages of the participating 
students; the characteristics of the courses on which the students were enrolled ajid the 
professional qualifications and experience of the lecturers taking pan in the pilot 
study. This level of detail was required before the validity of the factor analysis could 
be confirmed. 
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Initial Feedback 

A meeting was arranged with Professor Janssen in Belgium in Septeinoer 1992 to 
receive the initial feedback on the results of the Pilot Study. The results confirmed 
the validity of the modified questionnaire as a suitable tool for evaluating lecturing 
behaviour in an educational system outwith Belgium and for providing data that could 
be readUy compared with similar data produced in relation to Leuven University. 

So far as the evaluations of the individual lecturers were concerned Professor Janssen 
expressed some surprise at the high scores achieved by the Napier lecturers across the 
board. 

Feedback for the individual lecttirers was provided and included with a paper giving 
a brief explanation of the aims and purpose of the questionnaire, and guidelines on 
how the results should be interpreted. These results were passed to the lecmrers 
concerned in sealed envelopes with a covering letter inviting any who wished to do 
so to discuss their results privately with Professor McLachlaii. 

ESTABLISHMENT OF THREE YEAR COLLABORATT/E RESEARCH PROJEC T 

As a result of the successful completion of the pilot study and of the confimiation of 
the validity of the modified questionnaire as a tool for evaluating lecturing behaviour 
a three year collaborative research project has now been established between Napier 
University and Leuven University. The first year of the project hsis been completed 
with nine lecturers volunteering to take part in the evaluation involving 434 student 
responses, which was completed in early April 1993. This time, whUe all of the 
lecturers taught on Degree and Higher National Diploma courses some taught first 
year students and some taught second year students. 

The same procedures were adopted whereby the completed optically read answer 
sheets, coded to identify individual lecturers, were sent to Leuven for analysis. 
However Professors Janssen and Verhaeghe visited Napier in June 1993 to give a 
seminar to interested staff on their research activities in general and on the 
collaborative research in particular. 

This seminar was followed by private interview with the participating lecturers to 
provide them with the results of the evaluation, to discuss their scores and to identify 
any areas requiring particular attention. 

There will be a return visit to Leuven in the Autumn to discuss initial comparison of 
results between Leuven and Napier and to progress the project. A similar pattern of 
two visits per year will be adopted for the duration of the project with lecturers being 
interviewed to provide them with feedback in relation to their own evaluations. 
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1 1 . ANALYSIS OF RESULTS 1992 AND 1993 



11.1 Guidelines for Interpreting Results 



Given the 6-point scale adopted for completion of the questionnaire, and the fact that 
the point which differentiates an acceptable score from an unacceptable score is 3.5, 
mean scores should be interpreted in accordance with the following guidelines: 



Mean Score 


Category 


Students' Opinion 


EVALEC Interpretation 


LO - 1.49 


1 


Strongly disagree 


Highly unacceptable 


1.5 - 2.49 


2 


Disagree 


(Very) poor 


2,5 - 3.49 


3 


Mildly disagree 


Unacceptable 


3.5 - 4.49 


4 


Mildly agree 


Acceptable 


4.5 - 5.49 


5 


Agree 


(Very) good 


5.5 - 6.00 


6 


Strongly agree 


Excellent 



When interpreting the results as presented in the matrix attention should first of all be 
paid to the five student perception scores - the totals of the five columns. If a score 
is below standard or comparatively low, the individual cells ot tiiat column will reveal 
the specific lecturing strategy requiring attention. Improvement of that strategy will 
in turn improve the related lecturing behaviour. 

The Total row scores give more general information regarding the teaching 
activities/techniques which have been displayed - to a greater or lesser degree - and 
make it possible to identify priority areas where increased effort is required if 
lecturing behaviour is to be improved. 
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11.2 Mean Scores Achieved by Napier Lecturers in the Pilot Study, 1992 



EVALEC-matrix: mean fcores for the pcicepHon dimnuions, ipecific cell scoies and teaching activit!etAechnitM)es 





diiecting 

na 15 
alfa = .91 


organising 

ns9 
alfa « .85 


communicating 

n= 15 
alfa - .88 


stimulating 

ns 13 
alfa = .89 


motivating 

nr:26 
alfa = .96 


total 


cUnfication 
of inaCtuctional 
aitm and objectives 

n — 

n SS O 

alpha « .83 


setting the 
subject in 
context of 
futuie objectives 
and goals 

4.36 




plotting the 
courK/Wfty in 
which topics 
will be covered 
and linked 

4,43 




dcmonitraiing 
relevance of 
subject content 
and of the subject 
to the student's 
ovetall development 

4,48 


4.44 


tecognition of 

students' 

bttckgrounds 

n = 12 
alpha « .89 


avoiding trups - 
allcfting to 
difficult areas 

4.44 




linking up with 
experience and 
knowledge 

4.39 


tuning in to the 
heterogeneity in 
the claaroofxV 
accepting different 
poiiVs of view 

4.42 


promoting intcrrst 
ar.d encouraging 
deeper inve^tigatioti 

4.40 


4.41 


pfftpaiatlon: 
content 

n = 13 
alpha s .88 


anchonng new 
material within 
overall wbject 
context 

4.29 


stnicturing the 
lecture/choice of 
content 

4.90 


clarifying expectations 
and views in relation 
to subject content 

4.04 


offering (a broad) 
perepective^inking 
the subject to the 
cause overall 

4.31 


thrxHigh careful 
pfeparation providing 
a logical fianieworic 
which engeiHleis 
creative thought 

4.66 


4.46 


pflepaialion: 
explanation 

n=: 12 
alpha = .85 


highlighting 
impoftant aspects/ 
topics 

4.60 


building up the 
argument/ 
relating topics 
in sequence 

4.78 


stimulating thinking 
explanations in an 

4.32 


offering clear 
of interest in own 
adaptable w&y 

4.30 


sharing eviderKe 

subject and making 
topks easy to 
^ndef^aiid 

3.94 


4.35 


pirpantion: 
media 

n 3 12 
alpha - .84 


suitability of 
coune material and 
illustrations 

4.70 


using the board 
effectively 

4.55 

using other media 
effectively 

4.37 


co-ordinating study 
materials with lecture 
contsnt 
4.06 

taking notes 
4.40 


using interesting 
examples to enliven 
the lectures 

3.91 


ufitn}C topical and 
relevant examples 
and illustmlions 
in a well balanced 
way 

4.35 


4.41 


interaction 

n := 11 
atplia = .85 




prepared (o deal 
with the reacrions 
of the students 
4.27 

using the voice 
a. so 


dealing with 
que&tions/^omments 

4. fit 


entering into 
discussion/answering 
que^iot)^ tAitwith 
the nuiterial to be 
learned 

4.20 


kecpini: cixitact ttnd 
interest ongoing 

4.4^ 


4 <K 


evaluation of 
asaessment 

ns 10 
alpha « .81 


clarificfttitM) ot 
t\ic a«scs««me!>t 
- format 

4.58 

• criteria/areas 
to be teamed 

4.53 




Inniiliftri^tng students 
with t\yc content of 
c xaminali('>as/other 

4.56 




CKplnintng llic specific* 
expectations in icl.itton 
t(Y tl)c rcqutrctncnts ol 
asscssnKnt 

4.42 


4.52 


Total 


4.49 


4.62 


4.38 


4.25 


4.42 


4.42 



n « number of statements involved 

alpha mual be .7 or over to be of statistical significance 
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11.3 Mean Scores Achieved by Napier Lecturers in the first year of the Collaborative 
Research Project; 1993 



BVALEC-nwirix: mein scores for the peicepeion dimensions, specific ceU scores and teaching activitieaAechniques 





ditecttng 
n= 15 
alfa= .91 


organiaing 

n = 9 
alfa = .85 


n= 15 
alfa « .88 


ns 13 
alfa >. 89 


iitmaV aiiiiiL 

n = 26 
alfa - .96 


total 


clarification 
of inatnictional 
alma and objectives 

nag 

alphas: .83 


setting the 
subject in 
context of 
future objectives 
and goals 

3.56 




plotting the 
cuune/way in 
which topics 
will be covered 

■no UfUkCQ 

2.83 




demonstrating 
relevance of 
subject content 
and of (he subject 

overall development 

3.93 


3.44 


recognition of 

studenta' 

backgrounds 

ns 12 
alpha > .89 


avoiding traps - 
alleiting to 
difficuh areas 

2.39 




linking up with 
experience and 
knowledge 

2.44 


tuning in to the 
heterogeneity in 
the clasroom/ 
accepting different 
points of view 

4.60 


promoting interest 
and encouragmg 
deeper investigation 

4.28 


3.43 


preparation: 
content 

n:= 13 
alplM = .88 


anchoring new 
material within 
overall subject 
context 

4.02 


stnjcturing the 
lectureA^hoice of 
content 

4.14 


clarif>ing expectations 
and views in relation 
to subject content 

2.90 


offering (a broad) 
perspective/linking 
the subject to the 
cause overall 

4.04 


through careful 
prepjuation pnividing 
a logical framework 
whi^'h engciKlrn 
creative thought 

4.34 


3.89 


preparation: 
explanation 

n> 12 
alpha « .85 


highlighting 
impottant aspects/ 
topics 

3.48 


building up the 
argument/ 
relating topica 
in sequence 

4.42 


stimulating thinking 
explanations in an 

3.39 


offering clear 
of interest in own 
adaptable way 

2.49 


sharing evidence 

subject and making 
topes easy to 
utklerMand 

3.38 


3.43 


preparation: 
media 

ns: 12 
alphas. 84 


fuitability of 
courae material and 
illustrations 

4.44 


using the boaid 
effectively 

4.79 

using other media 
effectively 

4.08 


co-ordinating study 
materials with kcture 
content 
2.54 

taking notes 
4.09 


uaing iolereating 
examples to enliven 
the lecturea 

4.79 


u.%ing topical and 
relevatit examples 
and illustrati<ins 
in a well balanced 
way 

4.44 


4.17 


interaction 
alpha :z 




prepaied to deal 
with the reactions 
of the students 

4.29 

using (lie vi^ice 


dealing with 

que stions/commenta 


entering into 
di9cuasioiV*ns ring 
questions ouewith 
the material to be 
learned 

3.76 


keeping contact and 
intere^ ongoing 

\.'H\ 




evaluation of 
assessment 

n« 10 
alpha -.81 


clariflcation of 
the assessment 

- formal 

- criteriVaitas 
to be learned 

4.50 




lamiliarising students 
with the ciwtcnt of 
c X am 1 nat i iMis/other 
.•Assessments 

3.21 




explaining iIk spcrific 
cxpcctfltiiWs in rclatuH 
to the reijuirenKiMs oi 
aKsessmciK 

3.90 


1 

4.00 


Total 


3.82 


4.00 


3.17 


3.94 


4.02 


3.75 



n X number of Htatementa involved 

alpha must be .7 or over to be of statistical significance 
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12. SUMMARY OF RESULTS 
12.1 Pilot Study 

The overall mean scores attained by the Napier lecturers in the Pilot Study were 
commended and some surprise was expressed by Leuven regarding their high level of 
scoring. No total mean score fell below the acceptable level - 3.5 and most were 
weighted towards the top of the acceptable range falling between 4.25 and 4.49. The 
two total scores for Assessment Evaluation and Organising Behaviour fell into the 
very good range at 4.52 and 4.62 respectively. 

So far as the means of individual cell scores were concerned 21 of the 33 scores were 
between 4.0 and 4.49 - at the top of the acceptable range - and 10 fell into the very 
good range between the scores of 4.53 and 4.9. The remaining two mean cell scores 
of 3.91 and 3.94 were well within the acceptable range. 

First Year of Collaborative Research Project 

All of the overall mean scores attained by the Napier lecturers were in the acceptable 
range (if 3.44 and 3.43 are interpreted as acceptable) except for one score of 3.17 for 
Communicating Behaviour. 

So far as the means of the individual cell scores were concerned 12 of the 33 scores 
were between 4 and 4.49 - at the top of the acceptable range - and 4 fell into the very 
good range between the scores of 4.5 and 4.79. Of the remaining 17 mean ceil scores 
7 were again within the acceptable range. However 6 mean cell scores fell into the 
unacceptable range and 4 fell into the (very) poor range with the lowest mean score 
recorded of 2.30. In other words 30% of the mean scores for individual cells were 
less than acceptable. 

12.3 Overall Summary 

The fact that all except one total mean scores achieved were positive scores, ie ui the 
acceptable range or above, indicated that no major remedial action was required in any 
given area. However, when the mean scores achieved by the individual lecturers ui 
1993 in individual cells were audied there was a pattern of lower score achievement 
across the board in relation to 'Communicating Behaviour*. In particular it was seen 
that some remedial action would have to be taken in relation to the extent to which 
lecturers explained their lecture plan in advance: linked their lecture content to the 
students' prior knowledge and experience: clarified their expectations iind personal 
views iji relation to the subject content; co-ordinated study materials with lecture 
content and stimulated thinking as a result of the way in which they explained their 
subject. In addition there was a tendency for lecturers to tell students that the 
assessments would be difficult and that they would have to woilc hard or fail rather 
than the adoption of an encouraging, helpfiil approach to build up the students' self 
confidence. 
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The appropriate lecturers, in the private interview situation, were alerted to these 
percei\eci weaknesses and encouraged to take positive steps to improve these areas of 
their teaching activities/techniques. 

PRELIMINARY CONCLUSIONS 

The results of the first two years of lecturer evaluation have been encouraging in so 
far as they have confirmed the general high quality of teaching being carried out at 
Napier. The fact that those lecturers involved in the project so far have received good 
evaluation reports has encouraged other lecturers to volunteer to be assessed in the 
second/third years of the project. For this type of evaluation to be successful it is 
essential that the lecturers trust those carrying out the evaluation to maintain the 
confidentiality of their reports and that results are discussed face to face to ensure that 
the scores are appropriately interpreted and action agreed in relation to any identified 
weaknesses. In addition, should a number of weaknesses come to light, remedial 
action should be taken in only one or two aspects at a time if significant improvement 
is to be made in any area. In initial discussion with Leuven regarding future 
comparisons between the results obtained in the two Universities the following cultural 
differences between the Belgian and Scottish educational systems will have to be 
borne in mind: 

(i) the nature of the institution: in the past Napier has placed a greater 
emphasis on teaching than on research while at Leuven it has been the 
opposite; 

(ii) the selection process: at Napier, in general, the better qualified students are 
enrolled following a competitive selection process while Leuven has to cope 
with the open access system operated in Belgium, the only Faculty setting an 
entrance examination being the Faculty of Applied Sciences for prospective 
engineers; 

(iii) the standard of entry: at Napier the educational standard of first year 
students on entry is, in the main, comparable as most enter on the strength of 
SCE or GCE qualifications - the product of nationally set curricula and 
examinations. There is no national education stracture/examination scheme in 
Belgium - each secondary school devising and assessing its own courses 
leading to a wide disparity in ability and in subjects covered on entry to 
University 

(iv) degree of motivation: in general students at Napier are reasonably well 
motivated - choosing to come to University because they want to do so - while 
at Leuven motivation of first year students is a real problem with many 
making use of their right of access to delay having to find employment for one 
year; 
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(v) approach to teaching/learning: Napier tends to use a student-ccntred/self 
study approach involving teaching/learning packages; directed study, timetabled 
tutorials and continuous assessment. This is not the case in Leuven where 
teaching is mainly devoted to the mass lecture approach and the only 
assessment is the end of year examination. 

14. FUTURE DEVELOPMENTS 

The initial evaluations have already highlighted differences in the scores achieved at 
Napier and at Leuven - not only in the levels of the scores attained but also in the 
areas of comparative strengths and weaknesses. 

Evaluation will continue in 1994 and 1995 and steps will be taken later this year to 
begin a stmctiured comparison of results, supported by fimding provided by the 
Ministry of the Flemish Community in Belgium. 
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